There have been increased demands of tracking a human movement with a video camera in a restricted area around a gate or the like and identifying the human movement. For this purpose, a prior art tracking device is generally configured to determine a moving object area within each of monitored images by differentiation of the time series monitored images, and to enlarge thus determined moving object area. However, thus determined moving object area becomes smaller as the moving object moves faster, which renders the enlarged image of the moving object area only at a poor resolution. In order to settle this problem, another scheme of determining the moving object area is proposed in a paper “Human Tracking Using Temporal Averaging Silhouette with an Active Camera”, vol. J88-D-II, No. 2, pp 291-301 of the Transactions of the Institute of Electronics, Information and Communication Engineers ISSN:09151923, published on 2005 May 1. The paper proposes to determine the moving object area based upon motion vectors (optical flows) obtained for selected points in each of the consecutive monitored images. Initially, the scheme obtains a detection frame which is detected by background differentiation to surround the moving object detected by the image differentiation. Then, the scheme goes to obtain the motion vectors with respect to areas selected respectively inside and outside of the detection frame for each of the two consecutive images, discriminates the moving object area from the background based upon the analysis of the motion vectors, and extracts a silhouette of the moving object in the current image to determine a shape and a center of the moving object. Although this scheme is found effective in a circumstance where only one moving object is expected, it is rather difficult to identify the moving object where more than one moving object is expected in a field of view of the camera. In order to alleviate this disadvantage, it may be thought effective to rely upon a template by which a target moving object could be discriminated from the other moving object. Since the silhouette of the moving object is defined as a collection of parts having the same motion vectors, a precise shape of the moving object is rather difficult to be extracted. Consequently, the above scheme is not satisfactory to realize a reliable determination of the moving object even with the addition of the template.